Method and system for designing adaptive, diagnostic assessments

ABSTRACT

A method and system for administering an assessment to a student are disclosed. The expected weight of evidence may be calculated for each of one or more tasks based on a student model pertaining to a student. A task may be selected based on the calculated expected weights of evidence. The selected task may be administered to the student, and evidence may be collected regarding the selected task. The student model pertaining to the student may be updated based on the evidence. A determination of whether additional information is required to assess the student may be made. If additional information is required to assess the student, the above steps may be repeated. Otherwise, a proficiency status may be assigned to the student based on the student model.

CROSS-REFERENCE TO RELATED APPLICATIONS

This application claims priority to, and incorporates herein byreference in its entirety, U.S. Provisional Application No. 60/654,982,entitled “Designing Adaptive, Diagnostic Math Assessments for Sightedand Visually Disabled Students” and filed on Feb. 22, 2005.

BACKGROUND

In the United States, student difficulties in mathematics tend to emergein middle school. For example, the results from a study regarding trendsin international mathematics and science indicate that while U.S. fourthgraders perform above the international average in mathematics, U.S.eighth-grade students perform at or below the international average. Bythe end of high school, U.S. students perform far below theinternational average.

In part, this downward trend among U.S. students may result from a shiftin the content that is being presented. Until the fourth grade,mathematics focuses on arithmetic instruction. In middle school, themathematics curriculum typically becomes more visual (e.g., studentslearn to interpret and construct graphs) and more abstract (e.g.,students learn to interpret and represent algebraic expressions).

One problem with current teaching methods is that by the time results ofhigh-stakes accountability tests are disseminated, classroom teachingmethods cannot generally be changed to address weak areas ormisconceptions of students. For example, if students in a particularclass have difficulty understanding and applying the quadratic equationand such deficiency and/or misconception is discovered upon theadministration of a high-stakes examination or an examination presentedat the end of a semester or other grading period, the ability of theteacher to receive and comprehend the results and incorporate thisknowledge into a lesson plan is difficult given an established coursecurriculum. In contrast, determining that the deficiency and/ormisconception exists while the material is being taught could permitadditional or varied instruction to be provided in a classroom setting.Accordingly, enhancing student learning of mathematics material that ismore visual and more abstract may permit students to actively solveproblems and receive timely diagnostic feedback that can further thelearning process.

In addition, some students can be heavily impacted by the emphasis ongraphic and/or abstract mathematics. For example, the increased visualnature of the content can provide a distinct disadvantage to studentsthat are interested in mathematics, but have visual disabilities.

Presenting alternative representations of the same or similar conceptsin tasks, examples, and the like can augment comprehension andaccommodate various disabilities. For example, when transforming contentfrom a visual format to an auditory format, it is important to providerepresentations that convey the same meaning. In this manner, no studentis unfairly advantaged or disadvantaged because of the format of theassessment task. For example, the notion of providing equivalentrepresentations is a central requirement of the World Wide WebConsortium's (W3C) Web Content Accessibility Guidelines. Under theseguidelines, Web content authors provide text equivalents or textdescriptions for non-text content (images, audio, video, animations,etc.).

Such text equivalents are rendered as visually displayed text, audioand/or Braille. Furthermore, audio presentations are carried out byhaving the text description read aloud via a live reader, pre-recordedaudio or synthesized speech. However, the use of a text descriptionrendered in audio to convey the meaning of a graph for a person who isblind can be confusing. Such an audio representation can exceed certainof the test taker's cognitive capacities. For example, a textrepresentation of FIG. 1 could read as follows:

-   -   This figure shows a straight line drawn on a two-axis system,        with a horizontal axis labeled X and a vertical axis labeled Y.        All four quadrants are shown. The line begins in the third        quadrant and moves upward and to the right; it crosses the        negative X-axis, passes through the second quadrant, crosses the        positive Y-axis, and ends in the first quadrant. Three points        are shown, two on the line and one in the fourth quadrant. The        point on the line in the first quadrant is labeled X, Y; the        point on the line in the third quadrant is labeled X-sub-one,        Y-sub-one. The point in the fourth quadrant is labeled X,        Y-sub-one. In addition, two dashed line segments are shown, one        that drops vertically from the point X, Y and connects it to the        point X, Y-sub-one, and one that moves horizontally to the right        from the point X-sub-one, Y-sub-one and connects it to the point        X, Y-sub-one. This forms a right triangle with the solid line as        a hypotenuse, the horizontal dashed line as the base, and the        vertical dashed line as a side.

Navigating through the audio presentation can be cumbersome, regardlessof whether, for example, a live reader is asked to repeat portions ofthe presentation or a pre-recorded audio presentation is navigated froma cassette tape. However, improvements can be obtained. The student canbe allowed to control the rate of speech and to navigate through thecontent in different ways (e.g., sentence by sentence or word by word).A pre-recorded audio presentation can be similarly improved over anaudiocassette by providing similar navigation capabilities, such asthrough a digital talking book technology. If the student reads Braille,the text description of the graphic can be conveyed via Braille ineither a hard copy or refreshable format.

However, a limitation of all of these approaches is that they merelyprovide access to the text description of the graphic rather than to thegraphic itself.

What is needed is a system and method of applying an evidence-centereddesign (ECD) approach to task development to further the learningprocess.

A need exists for an adaptive algorithm for task selection that can beused with an ECD system.

A need exists for a system and method of providing assessment services,adaptive e-learning and diagnostic reports.

A further need exists for a system and method that provides reasonableaccommodations to students that would otherwise be prevented fromlearning or being assessed due to the nature of the particular subjectmatter.

The present disclosure is directed to solving one or more of theabove-listed problems.

SUMMARY

Before the present methods, systems and materials are described, it isto be understood that this disclosure is not limited to the particularmethodologies, systems and materials described, as these may vary. It isalso to be understood that the terminology used in the description isfor the purpose of describing the particular versions or embodimentsonly, and is not intended to limit the scope.

It is also noted that as used herein and in the appended claims, thesingular forms “a,” “an,” and “the” include plural references unless thecontext clearly dictates otherwise. Thus, for example, reference to a“task” is a reference to one or more tasks and equivalents thereof knownto those skilled in the art, and so forth. Unless defined otherwise, alltechnical and scientific terms used herein have the same meanings ascommonly understood by one of ordinary skill in the art. Although anymethods, materials, and devices similar or equivalent to those describedherein can be used in the practice or testing of embodiments, thepreferred methods, materials, and devices are now described. Allpublications mentioned herein are incorporated by reference. Nothingherein is to be construed as an admission that the embodiments describedherein are not entitled to antedate such disclosure by virtue of priorinvention.

Enhancing student learning of mathematics material that is more visualand more abstract may permit students to actively solve problems andreceive timely diagnostic feedback. In addition, presenting alternativerepresentations of the same or similar concepts in tasks, examples, andthe like may augment comprehension and accommodate various disabilities.Adjusting learning environments and/or content to suit an individualstudent's needs may substantially improve learning as well.

In an embodiment, a method of administering an assessment to a studentmay include calculating the expected weight of evidence for each of oneor more tasks based on a student model pertaining to a student,selecting a task based on the calculated expected weights of evidence,administering the selected task to the student, collecting evidenceregarding the selected task, updating the student model pertaining tothe student based on the evidence, and determining whether additionalinformation is required to assess the student. If additional informationis required to assess the student, the above steps may be repeated toselect and administer a new task. Otherwise, a proficiency status may beassigned to the student based on the student model.

In an embodiment, a processor-readable storage medium may contain one ormore program instructions for performing a method of administering anassessment to a student. The method may include calculating the expectedweight of evidence for each of one or more tasks based on a studentmodel pertaining to a student, selecting a task based on the calculatedexpected weights of evidence, administering the selected task to thestudent, collecting evidence regarding the selected task, updating thestudent model pertaining to the student based on the evidence, anddetermining whether additional information is required to assess thestudent. If additional information is required to assess the student theabove steps may be repeated. Otherwise, a proficiency status may beassigned to the student based on the student model.

BRIEF DESCRIPTION OF THE DRAWINGS

Aspects, features, benefits and advantages of the embodiments describedherein will be apparent with regard to the following description,appended claims and accompanying drawings where:

FIG. 1 depicts a diagram used in an exemplary task.

FIG. 2 depicts relationships among the ECD models according to anembodiment.

FIG. 3 depicts a flow diagram for an exemplary method of determining anext task based on the expected weight of evidence according to anembodiment.

FIG. 4 depicts an exemplary student model according to an embodiment.

FIG. 5 depicts a tactile graphic for use as an exemplary accommodationaccording to an embodiment.

FIG. 6 depicts a flow diagram for the overall method of the inventionaccording to an embodiment.

FIG. 7 depicts a typical computer system including a processor, and anexemplary processor-readable storage medium according to an embodiment.

DETAILED DESCRIPTION

An “adaptation” or “adaptive capability” may include a system'scapability to adjust itself to suit particular characteristics of alearner and may include the customization of instructional material(e.g., content selection, sequencing and/or format) to suit differentlearner characteristics.

“E-learning” or “electronic learning” may include the delivery of anyinstructional and/or training program using one or more interactivecomputer-based technologies. E-learning may be used where networking ordistance communications are involved. For example, e-learning mayinclude, without limitation, distance learning and/or Web-basedlearning.

A “task” or an “item” may each include a question that elicits and/orprompts for an answer and/or a response.

Adjusting learning environments and/or content to suit an individualstudent's needs may substantially improve learning. Aptitude-treatmentinteraction (ATI) may be used to further a student's understanding ofmathematics material. In ATI, aptitude may refer to any individualcharacteristic that accounts for the level of student performance in agiven environment, and treatment may refer to the variations in, forexample, the pace, format and/or style of instruction. Differenttreatments may be more or less suited to different combinations ofstudent characteristics. For example, if it is known that a personcannot process visual information, but can hear well, and equivalentcontent is available in visual and auditory formats, ATI may recommendthat the content be delivered in the auditory format for that person.

Methods of customizing content may include determining what to present(referred to herein as microadaptation) and determining how to bestpresent it (referred to herein as macroadaptation). Microadaptation hasbeen a fairly elusive goal among educators for some time, as can be seenin Bloom, B. S., “Learning for Mastery,” Evaluation Comment, vol. 1(2),pp 1-12 (1968); Bloom, B. S., “The 2-Sigma Problem: The Search forMethods of Group Instruction as Effective as One-to-One Tutoring,”Educational Researcher, vol. 13(6) pp 4-16 (1984); and Tobias, S.,“Interest, Prior Knowledge, and Learning,” Review of EducationalResearch, vol. 64(1), pp 37-54 (1994). However, as described herein, anembodiment incorporating differential sequencing of content depending oneach learner's needs may be implemented using adaptive instructionaltechniques.

Microadaptation may be one method for customizing content.Microadaptation may include the real-time selection of content (i.e.,during the learning process) in response to a learner's inferredknowledge and skill state. Microadaptation may also be referred to asdomain-dependent adaptation. According to microadaptation principles,decisions about content selection may be based upon performance andsubsequent inferences of students' knowledge and skill states ascompared to the level that should have been achieved when instruction iscomplete. For example, if a student incorrectly solves a difficultassessment task pertaining to a particular concept or skill, a pluralityof alternatives may be indicated to increase the student's skill, suchas presenting new instructional material on the concept, administering aslightly easier assessment task directed to evaluating the sameproficiency, and the like. Alternatively, additional practice orremedial instruction may be warranted. When a student is believed tohave mastered a particular topic or otherwise achieved an “acceptable”level of performance, the student may be guided to new subject matter.

A second approach to adapting content may be macroadaptation, which mayinclude the customization of content according to more stable learnerqualities, such as cognitive or perceptual abilities. In contrast withmicroadaptation, macroadaptive decisions may be domain-independent andbased on learner information that is usually, but not always, collectedbefore instruction begins. Macroadaptation may relate to decisions aboutthe format and/or sequence of the content presented to the learner.Relevant learner information, such as cognitive variables, perceptualabilities, personality variables, and learning style, may be initiallycollected from a student. Subsequently, these data may be used to makeinformed decisions regarding the type of content or instructionalenvironment that is best suited to the individual.

An implementation that considers these two forms of adaptation may beused to substantially improve the learning process. Microadaptation maybe used to determine what to present to a learner and when to presentit. For example, a microadaptive algorithm may select an assessment taskthat provides the most additional information about a particular learnerat any given point in a learning and/or assessment process. In contrast,macroadaptation may be used to determine how it should be presented. Forexample, an assistive technology may be used to present mathematicalcontent to students with visual disabilities. Table 1 summarizes somegeneral differences between microadaptive and macroadaptive approaches.

TABLE 1 Alignment of Adaptation Type by Learner/System FeatureMicroadaptation Macroadaptation Feature (i.e., domain-dependent) (i.e.,domain-independent) Person System may adapt to fairly malleable Systemmay adapt to fairly stable Characteristic person characteristics such asperson characteristics such as knowledge, skills, and abilities thatcognitive variables, perceptual are the focus of instruction andabilities, personality variables, and assessment. learning style.Adaptive Microadaptive decisions may occur Macroadaptive decisions mayoccur Decision during instruction (through mainly prior to instruction(based on diagnostic assessment). pre-existing data sources or pre-instruction assessment). Consequence of Decision may affect what contentis Decision may affect how content is Adaptation presented (e.g.,determination of presented (e.g., differential when the student is readyto proceed sequencing or alternative presentation to the next part ofthe curriculum). format). Theoretical Adaptation may be based onAdaptation may be based on theory Underpinnings theoretical andempirical and research on ATIs, assessment information relating tolearning and validity and other information from pedagogical principlesthat provide individual learner differences. information about what toinstruct or assess and why.

As such, well-founded diagnostic assessments of proficiencies may bedeveloped. Good assessments may be used to obtain relevant informationthat permit inferences to be made regarding students' knowledge andskill states. Moreover, accurate inferences of current knowledge andskill states may support microadaptive decisions that promote learning.

Evidence-centered design (ECD) may attempt to obtain, among otherthings, clear answers to three basic assessment questions: (a) what isdesired to be determined about persons taking the assessment, (b) whatobservations (behaviors or work products) provide the best evidence forthese determinations, and (c) what kinds of tasks allow necessaryobservations to be made or pertinent evidence to be collected. Forexample, suppose a measure of students' knowledge of U.S. state capitalsis desired. Evidence of high proficiency may include a given studentcorrectly listing the names of all capital cities by state. Thisevidence may be obtained orally, on paper and/or via computer using freerecall and/or matching tasks. The ensuing score on this assessment maybe interpreted in relation to pre-established scoring rules.

In order to apply an ECD framework to the design of assessment tasks, asubject matter expert, such as a teacher or test developer, may create,for example, three models: (a) a student model, which may define therange and relationships of the knowledge and skills to be measured, (b)an evidence model, which may specify the performance data associatedwith these knowledge and skills for varying levels of mastery, and (c) atask model, which may define the features of task performance situationsthat may elicit relevant evidence.

FIG. 2 depicts relationships among the ECD models according to anembodiment. As shown in FIG. 2, assessment design may flow conceptuallyfrom student models through evidence models to task models, although theflow may be less linear and more iterative in practice. Conversely,diagnosis or inference may flow in the opposite direction. In otherwords, when a diagnostic assessment task is administered, the action(s)performed by a student during the solution process may provide evidencethat is analyzed by the evidence model. The results of this analysis mayinclude scores and/or other data that are communicated to the studentmodel to update relevant proficiencies. An adaptive algorithm may beinvoked to select a new task to be presented to the student based on theupdated proficiency values in the corresponding student model. The cyclemay repeat until the tasks are completed, time has run out, mastery hasbeen achieved and/or some other termination criterion has been met.

In this manner, a psychometrically sound approach for designingassessments and modeling student performance may be provided. The ECDapproach may provide a framework for developing assessment tasks thatare explicitly linked to claims about learner proficiencies via anevidentiary chain.

A student model may refer to a record of what a student is believed toknow and/or not know in relation to some referent knowledge and skillmap, which may be referred to as a proficiency model. A student modelmay be modeled using a Bayesian inference network (BIN). BINs may beemployed to represent, monitor and update the student model and tocompute probabilistic estimates of proficiency (e.g., the probabilitythat a student has a “very strong” grasp of a particular concept may be95%) at various points in time. A Bayesian approach to student modelingmay be used in an e-learning system to inform microadaptivedecisions—enabling the system to choose the best piece of content, suchas the most helpful and informative assessment task, to present next.

An evidence model may be described in relation to the observablefeatures of students' work products (or behaviors) that constituteevidence about proficiencies. Proficiencies may be represented as nodesor variables in the student model. Thus, evidence models may attempt todetermine which behaviors and/or performances reveal targetedproficiencies, and what connections exists between those behaviors andthe student model variables. An evidence model may thus define anargument regarding why and how the observations in a given tasksituation (i.e., student performance data) constitute evidence aboutstudent model variables. For example, an evidence model may assist indetermining what is known about a student's “knowledge of U.S. statecapitals” if the student can freely recall 40 of the 50 state capitals.The evidence model may also assist in determining whether such aperformance is better or worse than matching 48 capitals to theirappropriate state when each is displayed.

Evidence models may include evidence rules and statistical sub-models.An evidence rule may determine how the results of a given performanceare extracted from (or identified in) a particular work product. Thus,evidence rules may emphasize how the student performs or responds. Astatistical sub-model may express how the observable variables depend onor link to student model variables. As such, statistical sub-models maylink the extracted data to targeted proficiencies denoting what thestudent knows and how well the student is believed to know it.

A given work product may yield one or more observable variables. Forexample, if a student writes a short essay, the essay may become thework product for a writing assessment task and may be evaluated in termsof various proficiencies, such as spelling, grammar, syntax and/orsemantics. These proficiencies may be assessed and updated individuallyand/or may be considered as a more general “writing skills” proficiency.Accordingly, the evidence rules may differ to focus on individual orholistic rubrics. An exemplary holistic evidence rule for “highlyproficient” writing may include: “The essay is clear and concise, withperfect spelling; and no grammar, syntax or semantic errors present.”

Evidence models may thus represent an evidentiary chain between tasksand proficiencies. Moreover, a necessary condition for an evidence modelmay be that it shares the same work-product specifications as aparticular task model. In other words, what the student produces in thetask situation and what the evidence rules examine may be required to bethe same.

Tasks may be the most obvious part of an assessment and may be used toelicit evidence (observables) about proficiencies (unobservables). Atask model may provide a framework for describing the situations inwhich students act in terms of, for example, (a) the variables used todescribe key features of a task, such as content, difficulty, and thelike, (b) the presentation format, such as directions, stimuli, prompts,and the like, and (c) the specific work or response products, such asanswers, work samples, and the like. As such, task specifications mayestablish what a student is asked to do, what kinds of responses arepermitted, what types of formats are available, whether the student willbe timed, what tools are allowed (e.g., calculators, dictionaries, wordprocessors, etc.), and the like. Multiple task models may be employed ina given assessment.

Different task models may produce different tasks, which may vary alonga number of dimensions (e.g., media type and difficulty level). Forexample, the following three tasks may define three levels of difficultyin a student model variable: “Find the common difference in anarithmetic sequence:”

-   -   EASY—Find the common difference for the following arithmetic        sequence:    -   1, 7, 13, 19, 25, . . . Enter answer here: _(——————)    -   INTERMEDIATE—Find the common difference for the following        arithmetic sequence:    -   0.00, 0.49, 0.98, 1.47, 1.96, . . . Enter answer here: _(——————)    -   DIFFICULT—Find the common difference for the following        arithmetic sequence:    -   0.03, 0.95, 1.87, 2.79, 3.71, . . . Enter answer here: _(——————)

Note that the relationship between student model variables and taskssuch as those listed above may be that student model variables representthe concepts or skills being examined. The online manifestations ofthose variables may be the assessment tasks with which students interactand that elicit evidence about the variables. Thus, student modelvariables may be assessed (and their states inferred) in relation to alearner's performance on relevant tasks.

In an embodiment, the student model may be represented as a BIN. In anembodiment, one or more student model variables may have probabilitiesfor each of, for example, three proficiency level states: low, medium,and high. For example, a student who struggles with a specific conceptor skill (e.g., knows U.S. state capitals) may have the followingprobability distribution assigned to this variable: low (p=0.85), medium(p=0.10), high (p=0.05). More or fewer proficiency level states may beused for each student model variable within the scope of this disclosureas will be apparent to those of ordinary skill in the art.

In an embodiment, additional nodes may be used to provide granulatedinformation regarding a student's abilities. For example, if knowingeach state and its capital were each targeted as being important, fiftyadditional nodes may be represented (i.e., one per state, residing underthe parent node: “knows U.S. state capitals”). In an embodiment, otherproficiency level states may exist between the individual states and theglobal (parent) node as well. For example, additional nodes may be usedto assess students' knowledge of state capitals by region (e.g.,“mid-Atlantic states,” “New England states”). The student model may beused to reflect this hierarchy, and evidence may be collected andincluded at each corresponding proficiency level state to answerquestions regarding the student's understanding of the subject matter.Each variable may include its own probability distribution. For thedistribution described above (low=0.85, medium=0.10, high=0.05), thedistribution may be interpreted to mean, “It is likely this studentcurrently does not know all of the U.S. state capitals.”

Such probability distributions may be dynamically updated based on thecurrent, specific performance data (evidence) that influence the studentmodel. Maintaining an updated record of proficiency levels may helpdetermine proper interventions. For example, students performing lowerthan expectations (students having a high probability of a lowproficiency level) may benefit from remedial instruction; studentsperforming consistently with expectations (students having a highprobability of a medium proficiency level) may need to continuepracticing the current skill/concept; and those performing higher thanexpectations (students having a high probability of a high proficiencylevel) may be ready to move to more advanced material. However, a moreconcrete method for determining the most suitable task to next presentto a learner at a given time may be determined.

In an embodiment, the next task to be selected may be the task for whichthe expected weight of evidence is maximized. The expected weight ofevidence (WE) may be defined as:

${{WE}\mspace{11mu}\left( {H\text{:}T} \right)} = {\sum\limits_{j = 1}^{n}{{\log\left\lbrack \frac{P\left( {t_{j}❘h} \right)}{P\left( {t_{j}❘\overset{\_}{h}} \right)} \right\rbrack}{{P\left( {t_{j}❘h} \right)}.}}}$Here, T may refer to a task performance, and H may refer to the mainhypothesis. Either the main hypothesis is true (h) or the alternativehypothesis is true ( h). The variable n may refer to the number ofpossible outcomes for each task. In an embodiment, two possible outcomesmay exist for each task: correct or incorrect. Other embodiments mayinclude a plurality of possible outcomes within the scope of thisdisclosure. The variable j may represent the outcome index for aparticular task, and the variable t_(j) may be the value of the outcome.

In an embodiment, the weight of evidence for a particular task outcomemay be the log-odds ratio of the probability that a particular outcomewill occur given that the hypothesis is true, to the probability thatthe same outcome will occur given that the alternative hypothesis istrue. Thus, the expected weight of evidence, WE(H:T), for a particulartask may be the average weight of evidence across possible taskoutcomes.

With respect to the earlier example, when an instructional unit on U.S.state capitals has been completed, an assessment may be administered todetermine whether the students demonstrate high levels of proficiency ontasks assessing relevant content. A hypothesis of interest (h) may bethat the students are high on their state capital proficiencies, and thealternative hypothesis ( h) may be that they are not high.

In an embodiment, each student may take the assessment one task at atime. In an embodiment, upon the completion of each task by a student,two possible outcomes may exist: either the student solved it correctlyor incorrectly (t_(j)=1 or 0). Tasks may be rank-ordered based on thedifficulty levels for all of the tasks. The difficulty levels may bebased on, for example, familiarity, frequency and/or saliency data. Forexample, if the assessment were administered in New Jersey, an easy itemmay include identifying Trenton as New Jersey's state capital. A moredifficult item may include, for example, identifying the capital ofSouth Dakota.

Determining a proper question to ask first may depend upon the goal ofthe assessment. For example, if the goal of the assessment is todetermine whether the material has been mastered by a majority of thestudents, asking a particularly easy question that each student islikely to answer correctly may not provide additional informationregarding the students' proficiency levels. Accordingly, it may bedesirable to pose a more difficult question. Determining whether anadditional question should be posed to a student and, if so, thedifficulty level of such a question may be based on the student modelproficiency levels for the particular student, as updated based on theoutcome of the posed question, and on the one or more goals of theassessment as a whole.

On the basis of each outcome event, and in conjunction with thedifficulty of the current task and the current proficiency level valuesin the student model, which are unique to each student based on theirresponses and any prior information that had been received by the model,the WE may be calculated for the remaining set of assessment tasks.Accordingly, the next task selected (if any) may be the task that hasthe highest WE value (i.e., the task providing the most information inrelation to the specific hypothesis).

For example, if a student has a low proficiency level and misses adifficult item pertaining to the proficiency, the next task that may beselected (via the WE calculation) may be one directed to assessing thesame proficiency, but including an easier representation. For example,in the example described above, the student may initially be asked torecall the capital of South Dakota in response to an open-ended prompt(i.e., “What is the capital of South Dakota?”). This may represent adifficult task. If the student answers incorrectly, the student may bepresented with an easier, forced-choice variant, such as, “Which city isthe capital of South Dakota: (a) San Francisco, (b) Pierre, (c)Baltimore?”

Using WE may have advantages of being multidimensional, dynamic andflexible. In other words, WE may work with multidimensional BINs andallow estimation of a variety of student model variables (rather thanbeing limited to a single, general proficiency). Moreover, the model fora particular student may evolve over time by updating its variableestimates in response to actual performance data. Finally, the WEapproach may allow specification of a hypothesis of interest as opposedto requiring a default or fixed hypothesis.

FIG. 3 depicts a flow diagram for an exemplary method of determining anext task based on the expected weight of evidence according to anembodiment. The weight of evidence may be calculated for each task. Thetask with, for example, the highest WE may be selected. The selectedtask may be administered to a student, and evidence may be collected. Inan embodiment, the evidence may include the response to the selectedtask, other information pertaining to the task and/or to the studentand/or any other relevant information. The response may be scored basedon a heuristic. The student model, such as a BIN, may be updated toinclude the received information and/or evidence. It may be determinedwhether obtaining additional information would be beneficial toassessing the proficiency level of a student. If additional tasks wouldbe beneficial, the process may repeat by calculating the weight ofevidence for each remaining task (i.e., each task that has not alreadybeen administered to the student). Otherwise, the process may terminate.Termination may also occur if a threshold is exceeded, if time runs outand/or if no more tasks remain for assessing proficiency.

In an embodiment, two stages may characterize the design of an ECD-basedassessment: domain analysis and domain modeling. Domain analysis mayinclude a process of identifying, collecting, organizing and/orrepresenting the relevant information in a domain based on informationreceived from domain experts, underlying theory, supplementary materialand the like. In domain modeling, relationships may be established amongone or more student proficiencies, the evidence for the one or moreproficiencies and/or the kinds of tasks that elicit relevant evidence.Graphic representations and schema may be used to convey complexrelationships.

In an embodiment, the domain analysis phase may include considering therange of constructs that may be measured by the assessment. Relevantconstructs may be identified via expert practitioners, supportingmaterials, research articles, state and national testing standard and/orpractical requirements and constraints. For example, when designing anassessment that covers eighth-grade mathematics, teachers teachingstudents at that grade level may be consulted to determine theappropriate subject matter for the assessment. In an embodiment, apractical constraint may include limiting the scope of the assessment to2-3 weeks of material, which may correspond to the approximate length oftime that most teachers will spend on a classroom unit of instruction.

In an embodiment, “sequences as patterns” may be selected as a topic foran assessment. Prerequisites for the subject and the requisite skills toassess may be determined. Sample tasks and supplementary materials maybe developed to assist in designing the instructional unit. Further, adetermination of the proficiencies that may be appropriate to include ona pretest and/or an interim test designed for the instructional unit onsequences may be developed.

Once the breadth and depth of the proficiencies to test are determined,domain modeling may be performed. In the domain modeling phase,assessment designers may use information from the domain analyses toestablish relationships among proficiencies, tasks and evidence. Thedesigners may develop high-level sketches of the interrelationship amongthe proficiencies that are consistent with what they have learned aboutthe domain. Ultimately, the designers may create graphic representationsto convey these complex relationships. The designers may further developprototypes to test assumptions.

Key proficiencies and the manner in which they should be linked andorganized may be determined for a student model. For example, a graphicrepresentation may be created defining links between proficiencies. Oncethe student model is established, the evidence and task models may bedefined. FIG. 4 depicts an exemplary student model according to anembodiment. Features of the student model depicted in FIG. 4 may includethe following: 1) the model may be hierarchical. Each child node mayinclude only one parent node. 2) The root node that represents theproficiency, sequences as patterns, may have three child nodes. Eachnode may correspond to a different sequence type. 3) The proficienciesunder each sequence type in FIG. 4 may be identical except that noanalog may exist for common difference (arithmetic) or common ratio(geometric) in other recursive sequences. This may be because the otherrecursive sequences proficiency may be more broadly defined and maypertain to sequences taught at the eighth-grade level that arerecursively defined but are neither arithmetic nor geometric. Examplesof other sequences may include Fibonacci numbers, triangular numbers,and simple repeating patterns. Non-hierarchical relationships, differentnumbers of child nodes per parent node and/or different proficienciesamong child nodes may be implemented in a student model within the scopeof this disclosure. In other words, FIG. 4 is merely exemplary of astudent model and not limiting on the scope of this disclosure, whichincludes the embodiment shown in FIG. 4 and numerous other embodiments.

Brief descriptions of exemplary student proficiencies are provided inTable 2 below. In an embodiment, three levels of proficiency (e.g., low,medium and high) may be associated with each student variable. For eachproficiency level of each student model variable, a claim may bespecified describing what the student should know and be able to do. Anexemplary claim for a student with a high level of proficiency atfinding explicit formulas for geometric sequences (i.e., the nodelabeled explicit in the geometric branch of the student model of FIG. 4)may include: “The student can correctly generate or recognize theexplicit formula for the n^(th) term in a geometric sequence. Thestudent can do this in more challenging situations, for example, whenthe signs of the terms in the sequence are alternating, or when thestarting term and the common ratio are unequal.”

TABLE 2 Example Proficiency Descriptions Tree level Name in tree Fullname Description 1 Arithmetic Solve problems with A student with thisset of proficiencies can arithmetic sequences work with arithmeticsequences at the eighth-grade level. An arithmetic sequence may bedefined by a starting term a₁ and a common difference, d. The terms ofan arithmetic sequence may be as follows: a₁, a₁ + d, a₁ + 2d, a₁ + 3d,. . . , a₁ + (n − 1)d 2 Pictorial Represent pictorial A student withthis set of proficiencies can patterns as sequences interpret a graphic(e.g., a succession of (arithmetic, patterns of dots) as a sequence of ageometric, other particular type. recursive) 3 Algebra rule Generate arule for a A student who has this skill can express sequence as a rulesof generating terms in a sequence function or algebraically; the rule inthis case takes the expression form of an algebraic expression.(arithmetic, geometric, other recursive) 4 Explicit Generate a formula Astudent with this proficiency can use an for the nth term of a algebraicexpression to represent the nth sequence (arithmetic, term of asequence. For example, 5 + 2(n − 1) geometric, other is an explicit rulefor the nth term of an recursive) arithmetic sequence with an initialterm of 5 and a common difference of 2. In general, an explicit rule forthe nth term of an arithmetic sequence is: a_(n) = a₁ + (n − 1)d (whered is the common difference) and an explicit rule for the nth term of ageometric sequence is: a_(n) = a₁r^(n−1) (where r is the common ratio).

As described earlier, the evidence model may specify behaviors thatindicate the level of mastery associated with a particular proficiency.The evidence model may include, for example, two parts: evidence rulesand a statistical sub-model. The evidence rules may be characterized ateach of the three levels, per proficiency. Evidence associated with eachlevel for two proficiencies is shown in Table 3.

TABLE 3 Evidence Rules Specified for Two Sample Proficiencies, at EachLevel of Mastery Evidence Rules for High Evidence Rules for MediumEvidence Rules for Proficiency Proficiency Level Proficiency Level LowProficiency Level Represent The student can produce a The studentrecognizes that The student does not pictorial pattern that representsan the pictorial patterns have infer any mathematical patterns asarithmetic sequence, can mathematical significance, significance fromthe arithmetic recognize arithmetic but cannot consistently pictorialpatterns. sequences sequences represented as explain how or why.pictorial patterns, and can recognize the equivalence between numericand pictorial representations. Generate The student can generate Thestudent generates The student generates and justify geometric sequences.If a something that may be a something that does examples of list ofterms is given, all sequence but not necessarily not express a sequencegeometric terms in the sequence are a geometric sequence, or orgenerates a sequences correct. If a formula is generates a sequence thatis sequence that does not given, it is well formed geometric but hassome include a and correctly specifies an incorrect terms due tomultiplicative appropriate example. arithmetic errors, or operation asat least generates a formula that is part of the rule. close toexpressing the correct sequence.

The statistical sub-model may define a set of probabilisticrelationships among the student model variables (nodes) and observables.Prior probabilities (priors) may be estimated for the parent node (i.e.,sequences as patterns). In cases where the prior distribution is notknown in advance, values of approximately 1/n may be assigned for eachof the n possible states (i.e., 0.33, 0.33 and 0.34 for 3 states). Thepriors may specify the probabilities that a student is in the low,medium and high states for the parent node proficiency.

In an embodiment, for each of the other nodes in the model, two valuesmay be entered. One value may be an indicator of the relative difficultyof the tasks associated with that particular node, and the other may bea correlation that indicates the strength of the relationship betweenthe node and its parent node. These values may be used to produce a setof conditional probability tables, where one table may exist for eachnode except for the root node. Because each node in the exemplaryembodiment has three levels associated with it, each conditionalprobability table may have nine probability estimates (3 parent nodelevels multiplied by 3 child node levels). For example, a cell in thetable associated with the “model” node under “arithmetic” sequences mayindicate the probability (expressed as a value between 0 and 1) forhigh-level proficiency for tasks of type “model” given a medium-levelproficiency for “arithmetic” sequences. Students with high proficiencylevels may be considered likely to solve both hard and easy tasks, whilestudents with low proficiency levels may be considered likely to solveonly easy tasks.

A task model may provide a specification of the types of tasks thatmeasure the behaviors described in the evidence model. The task modelmay describe the features for each type of task included in anassessment. For example, the task model may describe different itemtypes included in an assessment, the nature of the stimulus, the stemand/or the options (if any). The task model may also describe how thestudent is required to respond to each type of task. For example, amultiple choice item may require the student to select an option, whilea numeric entry item may require a student to enter a number instead. Anexemplary item may include the following: “Find the missing terms in thefollowing arithmetic sequence: 4.68, _(——————), _(——————), 13.74, 16.76,19.78.” The item type, the nature of the stem and/or the number ofresponses may be exemplary task model variables included in the taskmodel specification. The exemplary item above may be a numeric entryitem because the student is required to enter numbers rather thanselecting an option. Two responses may be required for the above item(one for each blank). As shown, the stem may include both numbers andtext, but no graphics. The stem may include one or more words, numbers,pictures and/or tables.

In an embodiment, a plurality of tasks may be included per proficiencyat each level of difficulty. In FIG. 4, the thirty-two proficiencies mayrepresent the children of the main nodes (i.e., Sequences as Patterns,Arithmetic, Geometric and Other Recursive sequences). Accordingly, iftwo tasks are included per proficiency at each level of difficulty, 192tasks (i.e., 32 proficiencies, multiplied by 3 levels and 2 tasks perlevel) are required for the particular embodiment shown in FIG. 4. Tasksmay be selected from previously generated task items or may be developedindependently. In an embodiment, tasks may be developed usingquantitative item models, such as the item models described below. In anembodiment, items may be automatically generated and formatted from theitem models using software designed for this purpose.

The term item model may refer to a class of content equivalent itemsthat describe an underlying problem structure and/or schema. Aquantitative item model may be a specification for a set of items thatshare a common mathematical structure. Items in a model may also shareone or more formats, variables and/or mathematical constraints. A set ofitem models may be used to define the task model for an assessment. Thevariables in a quantitative item model may specify the range ofpermissible values that may replace the variable in an individual item.The constraints in a quantitative item model may define mathematicalrelationships among the variables. The number of items described by anitem model may depend on how the variables and constraints have beendefined.

Once an item model is defined, instances that are described by the itemmodel may be automatically generated. A description of an item model maybe programmed into software that generates the instances. In addition toproviding an organized structure for item development, an automaticapproach to item generation may provide considerable practicaladvantages because the generating software may perform the necessarycomputations and format the items automatically. In an embodiment, ECDmay be used as the guiding framework to inform the structure of itemmodels.

Table 4 may depict a simplified example of an item model with two itemsthat could be generated using the model. This item model may generateeasy items that link to the “extend” node under “arithmetic” sequences.

TABLE 4 An Example of an Item Model and Two Items Model templateVariables and constraints Model Extend the arithmetic A1 is an integersequence by finding between 1 and 9, inclusive the next term: D is aninteger between 2 and 9, inclusive A1, A2, A3, . . . A2 = A1 + D A3 =A2 + D Key = A3 + D Example item 1 Extend the arithmetic A1 = 1 sequenceby finding D = 3 the next term: 1, 4, 7, . . . 4 = 1 + 3 7 = 4 + 3 10 =7 + 3 Example item 2 Extend the arithmetic A1 = 5 sequence by finding D= 9 the next term: 5, 14, 23, . . . 14 = 5 + 9 23 = 14 + 9 32 = 23 + 9

With respect to macroadaptation, an exemplary adaptation may includeaccommodating for visual disabilities, i.e., blindness and low vision.In an embodiment, content may normally be presented visually and mayrequire students to use, for example, a mouse, a keyboard and/or anotherinput device to answer, for example, single selection multiple-choiceitems. In an embodiment, students may be required to use a keyboardand/or another input device to answer, for example, numeric entry items.One or more accommodations for making test content accessible toindividuals with visual disabilities may be implemented. For example,individuals with low vision may use screen enlargement software, whichmay allow users to enlarge a portion of a display screen. Moreover,individuals who are completely blind or who are otherwise unable tobenefit from screen enlargement software may be able to access an audiorendering of content and/or tactile graphics (e.g., raised-linedrawings).

The usability of specific accommodations may be considered whendetermining the validity of test scores (i.e., the degree to whichaccumulated evidence and theory support specific interpretations of testscores entailed by proposed uses of a test) obtained under accommodatedconditions. For example, it may be important to ensure that theaccommodation is usable and overcomes one or more accessibilitybarriers. However, it may also be important to ensure that anaccommodation does not provide an unfair advantage for the person thatreceives the accommodation. For example, allowing a person with amath-related disability (e.g., dyscalculia) to use an electroniccalculator on a mathematics test may make the test accessible andusable; however, if the test is intended to measure mental computation,the electronic calculator accommodation may tend to provide an unfairadvantage for that person, thereby potentially invalidating the results.

An ECD-based validity framework may be used that closely examinesevidentiary arguments. Careful attention to the definition of theconstruct (e.g., skills or abilities that are or are not part of what isintended to be measured) may be required.

The exemplary “sequences as patterns” assessment may be used to measurecognitive abilities (e.g., reasoning and knowledge of various sequences)rather than assessing the senses of sight, hearing and/or touch. Assuch, it may not be unreasonable, for example, to provide accommodationsthat reduce or eliminate the requirements for sight (imposed by thevisually displayed text and graphics under standard testing conditions)and instead rely on other capabilities, such as hearing and touch, whendelivering test content.

Another relevant piece of evidence for this assertion may be that theability to decode (decipher words from characters) may not be consideredto be part of “knowledge of sequences.” If decoding were defined asbeing an essential part of that construct, use of an audio accommodationmay threaten the validity of the assessment; specifically, the audiopresentation may read whole words at a time thereby reducing oreliminating the need for the student to demonstrate their decodingability.

In an embodiment, ensuring valid assessment results may depend on aplurality of additional and/or alternate factors. For example, havingadequate practice and familiarization materials, adequate time and thelike may be required as accommodations.

In an embodiment, the ability to work quickly may not be essential to“understanding sequences as patterns.” Furthermore, a person who isblind and using tactile or audio-tactile graphics may be likely torequire more time to complete an assessment than a non-disabled personreceiving the test under standard conditions. Accordingly, extra testingtime may be an appropriate testing accommodation.

Audio rendering of content may be termed a “read-aloud” accommodationbecause it involves reading the content aloud to the student. Theaccommodation may be implemented via a live human reader, prerecordedhuman audio and/or synthesized speech. In an embodiment, the audiorendering may verbalize text content (i.e., straight text) and non-textcontent, such as images, audio and/or video/animations. As discussedabove, non-text content may be translated into text equivalents, whichseek to convey the same meaning as the non-text content through text. Anaudio rendering of a mathematics test may also include speciallyscripted descriptions of mathematical expressions and tables. If theaudio rendering has been crafted to convey all necessary content, aperson who is visually disabled may use it without relying on, forexample, tactile graphics. However, understanding graphical material(pictures, graphs, etc.) may be significantly easier when an audiodescription is supplemented with tactile graphics. Tactile graphics maybe printed or pressed onto paper or plastic and may be felt with thefingertips. Tactile graphics may include Braille labels. Hard copyBraille versions of test content may provide an alternate accommodation;however, many individuals who are blind do not read Braille or have verylimited Braille literacy.

In an embodiment, a hybrid method of access combining tactile graphicsand audio may be used. In such an audio-tactile graphics embodiment, thestudent may touch a specific location on a tactile graphic and hear adescription pertaining to that location. The student may quicklynavigate from location to location to hear as much or as little of thedescription as desired. Such audio-tactile graphics may facilitateaccess to graphics-intensive content. In an embodiment, a tactile tablet(such as the Talking Tactile Tablet made by Touch Graphics, Inc. of NewYork, N.Y.) may be used to implement a system using audio-tactilegraphics.

The tablet may provide audio (read-aloud), tactile and visualmodification capabilities. Such capabilities may be particularly usefulfor test content that uses graphics, tables and mathematicalexpressions, which are often difficult to convey via words alone.

Developing an application using a tactile tablet may require thedevelopment of a tactile graphic. In an embodiment, a tactile graphicmay be a sheet of hard plastic that uses raised lines and textures torepresent points, lines and regions of a graphic, such as is shown inFIG. 5. A special printing process may be used to print the graphicalmaterial in ink on the tactile graphic to assist visually disabledindividuals with some sight. In an embodiment, some features of thegraphic may by an external personal computer. A developer may specifythe active regions on the graphic in software and may map each activeregion to one or more prerecorded audio segments.

For example, a student using such a system may press on the angledepicted in the lower-right corner of FIG. 5 and hear the words “110degrees” in prerecorded audio. This may enable a student who has avisual impairment (or another disability that impairs processing ofvisually-rendered content) to receive specific and interactive audiodescriptions of content that would ordinarily be presented onlyvisually. A tactile tablet system may allow the student to navigatethrough the test and select an answer using tactile (raised-line)controls on the tablet. In an embodiment, a student using the tactiletablet system may only use a keyboard and/or other input device, forexample, when answering constructed-response items.

In an embodiment, the basic audio-tactile capabilities of the tactiletablet system may be augmented with capabilities designed to make thesystem suitable for achievement testing. For example, the system mayenable a test and item directions to be received, navigation between andwithin items to be performed, typed responses to be received (ifapplicable) and answers to be confirmed. Synthesized speech may permitstudents to hear an audio representation of a response as it is entered.

In an embodiment, the microadaptation and macroadaptation modules may beintegrated into a single system. For example, a microadaptationimplementation that selects content for presentation to a learner duringas part of an assessment may be integrated with a macroadaptation modulesuch as the tactile tablet. Accordingly, blind and/or other visuallydisabled learners may benefit from the use of an adaptive contentpresentation unit based on the student model as updated by responsesprovided by the learner. In an embodiment, different microadaptationand/or macroadaptation modules may be used. For example, a module thattranslates an assessment into a foreign language for non-native speakersmay be utilized as a macroadaptation module for an assessment.

FIG. 6 depicts a flow diagram for the overall method of the inventionaccording to an embodiment. In step 600, the embodiment determines ifall of one or more tasks have been performed, and if so terminatesexecution. Otherwise, the embodiment calculates in step 602 the expectedweight of evidence for the task based on a student model pertaining to astudent; the student model preferably comprises a Bayesian inferencenetwork. The embodiment selects in step 604 a task based on thecalculated expected weights of evidence, then administers the selectedtask to the student in step 606. In step 608, the embodiment collectsevidence regarding the selected task. In step 610, the embodimentupdates the student model pertaining to the student based on theevidence. In step 612 the embodiment determines whether additionalinformation is required to assess the student. The determining maycomprise determining whether a threshold has been passed, or determiningwhether a time limit has been exceeded, for example. If additionalinformation is required, the embodiment returns to step 602, otherwiseproceeds in step 614 to assign a proficiency status to the student basedon the student model. The proficiency status may comprise one or more ofa high level of proficiency, a medium level of proficiency, and a lowlevel of proficiency. The student model comprises one or more variables,wherein each variable corresponds to a proficiency for the student,wherein each variable includes a plurality of probabilities, whereineach probability corresponds to the likelihood that the student has aparticular proficiency level for the proficiency.

FIG. 7 depicts a typical computer system 70 including a processor, andan exemplary processor-readable storage medium 80 according to anembodiment. Computer system 70 conventionally includes a monitor 72, amain processor section 74, a keyboard 76, and a portable storage mediuminput mechanism (e.g. mini-disk drive) 78. Processor-readable storagemedium 80 may include but is not limited to a floppy disk, a mini-disk,a CD-ROM, a DVD-ROM, flash memory, data tape, and other conventionalstorage media as may be known to those of ordinary skill in the art.

It will be appreciated that various of the above-disclosed and otherfeatures and functions, or alternatives thereof, may be desirablycombined into many other different systems or applications. It will alsobe appreciated that various presently unforeseen or unanticipatedalternatives, modifications, variations or improvements therein may besubsequently made by those skilled in the art which are also intended tobe encompassed by the disclosed embodiments.

1. A computer-implemented method of administering an assessment to astudent, the method comprising: for one or more tasks, calculating anexpected weight of evidence using a computer processor for the taskbased on a student model pertaining to a particular student, wherein thestudent model comprises one or more variables, wherein each of the oneor more variables corresponds to a proficiency of the student and isbased on student specific information collected prior to the assessment,wherein each of the one or more variables includes a probability of aplurality of probabilities, wherein each of the plurality ofprobabilities corresponds to a likelihood that the student has aparticular proficiency level, wherein the expected weight of evidence iscalculated based on the one or more variables corresponding to theproficiency for the particular student and the student specificinformation collected prior to the assessment; selecting one of the oneor more tasks based on the calculated expected weights of evidence usingthe computer processor; administering the selected task to the student;collecting evidence regarding the selected task; updating the studentmodel pertaining to the student based on the evidence using the computerprocessor; determining whether additional information is required toassess the student using the computer processor; if so, repeating theabove steps; and if not, assigning a proficiency status to the studentbased on the student model using the computer processor.
 2. The methodof claim 1 wherein the evidence comprises a scored response to theselected task.
 3. The method of claim 1, further comprising: scoring aresponse to the selected task.
 4. The method of claim 1 wherein thestudent model comprises a Bayesian inference network.
 5. The method ofclaim 1 wherein determining whether additional information is requiredto assess the student comprises determining whether a threshold has beenpassed.
 6. The method of claim 1 wherein determining whether additionalinformation is required to assess the student comprises determiningwhether a time limit has been exceeded.
 7. The method of claim 1 whereina status of the proficiency comprises one or more of the following: ahigh level of proficiency; a medium level of proficiency; and a lowlevel of proficiency.
 8. A non-transitory computer readable storagemedium containing one or more program instructions for administering anassessment to a student, the programming instructions, when executed,causing a computer processor to execute steps comprising: for one ormore tasks, calculating an expected weight of evidence for the taskbased on a student model pertaining to a particular student, wherein thestudent model comprises one or more variables, wherein each of the oneore more variables corresponds to a proficiency of the student and isbased on student specific information collected prior to the assessment,wherein each of the one or more variables includes a probability of aplurality of probabilities, wherein each of the plurality ofprobabilities corresponds to a likelihood that the student has aparticular proficiency level, wherein the expected weight of evidence iscalculated based on the one or more variables corresponding to theproficiency for the particular student and the student specificinformation collected prior to the assessment; selecting one of the oneor more tasks based on the calculated expected weights of evidence;administering the selected task to the student; collecting evidenceregarding the selected task; updating the student model pertaining tothe student based on the evidence; determining whether additionalinformation is required to assess the student; if so, repeating theabove steps; and if not, assigning a proficiency status to the studentbased on the student model.
 9. The computer readable storage medium ofclaim 8 wherein the evidence comprises a scored response to the selectedtask.
 10. The computer readable storage medium of claim 8, furthercontaining one or more programming instructions for scoring a responseto the selected task.
 11. The computer readable storage medium of claim8 wherein the student model comprises a Bayesian inference network. 12.The computer readable storage medium of claim 8 wherein determiningwhether additional information is required to assess the studentcomprises one or more programming instructions for determining whether athreshold has been passed.
 13. The computer readable storage medium ofclaim 8 wherein determining whether additional information is requiredto assess the student comprises one or more programming instructions fordetermining whether a time limit has been exceeded.
 14. The computerreadable storage medium of claim 8 wherein a status of the proficiencycomprises one or more of the following: a high level of proficiency; amedium level of proficiency; and a low level of proficiency.
 15. Themethod of claim 1 wherein calculating the expected weight of evidencecomprises calculating$\sum\limits_{j = 1}^{n}{{\log\left\lbrack \frac{P\left( {t_{j}❘h} \right)}{P\left( {t_{j}❘\overset{\_}{h}} \right)} \right\rbrack}{P\left( {t_{j}❘h} \right)}}$wherein n is a number of potential outcomes for a particular task, j isan outcome index for the task, t_(j) is a value corresponding to outcomej, P(t_(j)|h) is a probability that the outcome occurs if a hypothesisis true, and P(t_(j)| h) is the probability that the outcome occurs ifthe hypothesis is false.
 16. The computer readable storage medium ofclaim 8 wherein calculating the expected weight of evidence comprisesone or more programming instructions for calculating$\sum\limits_{j = 1}^{n}{{\log\left\lbrack \frac{P\left( {t_{j}❘h} \right)}{P\left( {t_{j}❘\overset{\_}{h}} \right)} \right\rbrack}{P\left( {t_{j}❘h} \right)}}$wherein n is a number of potential outcomes for a particular task, j isan outcome index for the task, t_(j) is a value corresponding to outcomej, P(t_(j)|h) is a probability that the outcome occurs if a hypothesisis true, and P(t_(j)| h) is the probability that the outcome occurs ifthe hypothesis is false.
 17. The method of claim 1 comprisingcalculating expected weights of evidence for remaining tasks of theassessment based upon the updated student model, the student model beingupdated based upon evidence received in response to a question of theassessment.
 18. The method of claim 17 comprising selecting a next taskrelating to a same proficiency as a prior task, next task having adifferent representation than the prior task, the student model beingupdated based upon evidence associated with the prior task.
 19. Thecomputer readable storage medium of claim 8 comprising calculatingexpected weights of evidence for remaining tasks of the assessment basedupon the updated student model, the student model being updated basedupon evidence received in response to a question of the assessment. 20.The computer readable storage medium of claim 19, the processinginstructions, when executed, causing the processor to select a next taskrelating to a same proficiency as a prior task, next task having adifferent representation than the prior task, the student model beingupdated based upon evidence associated with the prior task.